Flash memory erase block sizes, from present manufacturers and for projected future flash memories, are not a single, fixed size. One flash memory may have an erase block size of 4 MB, another at 8 MB, 9 MB, 12 MB, 16 MB, 24 MB, 64 MB, etc., and soon flash memory will be available with 80 MB erase blocks. It is projected that an erase block size could be 1 GB in 3 to 10 years from today. This is not a problem for solid-state drives tied to a single vendor and/or single design of flash memory, with uniform block size. However, it is a problem for a storage cluster with the ability to add or replace solid state storage elements, and with a projected storage cluster lifespan over many years with an evolving heterogeneous mix of blades and solid-state memory dies.
A storage system may be built from multiple elements (which may be removable) that provide durable storage and present such to the storage system. For example, the durable storage may be built from multiple flash memory chips. Within that context, a storage system manages erase blocks within and across the durable storage elements that contain those flash memory chips, or at least maps segments of data managed by the storage system implementation directly onto erase blocks or ranges of erase blocks. Modifying software for every hardware replacement in such a storage system is one undesirable option or solution. Another undesirable option or solution is to use the smallest erase block size as a least common denominator for physical memory allocation and allow wasted memory space in the larger erase blocks. Another problem is that, if the size of a committed work load is determined by the erase block size, in those system areas where granularity of work is a data segment, this requires a much larger unit of work which must be completed before processing in a storage system implementation can move to another task that depends on completion of previous tasks. A very large erase block size could then adversely affect access times and latencies.
Yet another problem arises because, at boot time, the storage system needs to look up metadata, particularly information about which erase blocks are live and where erase blocks map into the address space of the system. One approach to dealing with variable sized erase blocks is to fragment the block into multiple pieces and hand the pieces out to different consumers. If the consumers write metadata at known offsets within their fragments, it can be difficult to find these fragments when booting the system—as the specification of the fragmentation may not be known yet. Increasing the number of places to look for such metadata (e.g., at 1 MB offsets instead of the top of an erase block) massively increases the load on the system at boot for such lookups. Therefore, there is a need in the art for a solution which overcomes the drawbacks described above.